1. Field of the Invention
This invention relates to a solid-state image pickup device (or a solid-state image sensor) used as a two-dimensional image sensor in object recognition devices, image information input devices, video cameras, and electronic still cameras, for both home and industrial uses.
2. Description of the Prior Art
As the result of progress in electronic technology, many of the electronic appliances introduced recently are provided with highly intelligent functions which make them practically independent systems, irrespective of whether they are intended for household or industrial applications. Generally, the systems minimally comprise input units, processor units, and output units, most also being equipped with converters at their input units to transform received external information into corresponding electrical signals. More specifically, the systems need sensors which can see, hear, taste, smell and feel by discerning objects as humans do with their five senses and converting the obtained physical and chemical information into corresponding electrical signals. And just as eyes are considered the most important of the human senses, visual sensors, in particular two-dimensional image sensors, which can take over part of the functions of human eyes, come first in the order of importance of all the sensors. To date, various types of two-dimensional image sensors have been developed. With the marked progress in the latest semiconductor technology, solid state two-dimensional image sensors utilizing semiconductors have appeared on the market. Especially noteworthy among them are those using the charge-coupled devices (hereafter called CCD's) which have contributed greatly to the technical developments in this field, finding wide application in both home and industry.
As a natural trend of the times, image pickup tubes (camera tubes), which now constitute the main stay in two-dimensional image sensors, are gradually being replaced by the new solid-state types. It is anticipated that solid-state image sensors will become dominant in the future, excepting special limited applications of image pickup tubes.
Moreover, the recent technical innovations in information processing and electronic circuitry techniques have opened up still newer fields of application for two-dimensional sensors. Example industrial applications can be seen in the object recognition devices used in robots and other automation equipment, in character recognition OA (office automation) devices of the automatic reading of processing parcels and in image information input devices which can take in at a glance full page contents of drafting paper.
In the home electronics category, the compact size and lightweight features of CCD and other new solid-state devices have expanded their use as image pickup elements in portable video cameras. In addition, these devices are now under review for use in burglar alarm systems as external monitors for interphones, making use of their advantages of such as compactness, long life, and less susceptibility to seizing.
It should be noted, however, that the greater the number of new applications for these solid-state image pickup devices, uses not conceived for conventional devices, the greater the probability of their failure to meet such new requirements in any of the many ramifications of such applications, if the new image pickup devices cannot be refined through improvement.
Some symptoms are beginning to surface in some of the new fields of application of such solid-state type image pickup devices, in view of the new requirements for them.
The first problem is that conventional solid state image pickup elements can only output picture element image information within a certain limited area of the overall image information receiving area. Although this has not yet become as a critical problem, it is well foreseen that future demand will require the ability to extract image information within any desired area of the overall receiving area of the solid-state image pickup device, in response to external control signals. In the following paragraphs, the situation calling for this function will be described.
In connection with the initial requirement for solid-state image pickup elements for industrial application, it goes without saying that the image recognition technique will become one of the very important techniques, for example, in robots and other intelligent equipment and appliances to be developed in future.
To obtain better object perception, objects to be perceived must be brought to and maintained at certain specific positions, as at the center of the screen, for instance; as a prerequisite, the objects must be within the area of the screen. To achieve this, it is sufficient to control the orientation of the image pickup unit or the direction of the optical axis in the direction of the centerline of the object, either mechanically or optico-mechanically. However, because of problems with conventional mechanical control systems, such as reduced reliability due to a greater likelihood of moving part failure and mechanical limitations on frequency response performance, in addition to limitations as to size, weight and power consumption, a purely electronic control method is to be used to obtain accurately directed pictures of the objects to be perceived, with the moving parts control system in present use functioning only as a supplemental control for rough positioning in the general direction of the object. Thus we foresee keen demand for image pickup devices capable of selectively picking up only the image within the area of the target objects covered on the screen in the current image pickup direction, selectively and purely electronically in response to external control signals. At the present stage, however, such needs have not yet been clarified as explicit requirements, as only limited aspects of this problem have been solved, by such techniques as tentatively storing image information in solid-state memory for later processing, without direct processing within the image pickup devices. Accordingly, there has not been much technical development with regard to outputting images within any desired area, in particular by using solid-state image pickup devices. As a result, there have been no established techniques in this regard upon which to draw.
Besides in application for object recognition in robots, this problem is becoming evident as well with character recognition OA devices for reading papers, as demand increases for random readout by electronic means, calling for output of the image information within unspecified partial areas, within the overall information provided on the screen, without mechanical movements of chart paper and/or image pickup optical systems.
Now, as to how the first requirement mentioned above poses a problem vis-a-vis solid-state image pickup devices used in home electronics products. One of the most familiar examples is the TV monitor unit used in homes or supermarkets for security purposes, in which mechanical means are employed to expand the range of surveillance by reciprocating motion at small amplitudes. In such installations, electronic control of image pickup ranges is better than the current system of changing the image pickup ranges mechanically, but such a function is not provided in conventional solid-state image pickup devices.
In recent years, video cameras have been diffused among general consumers at a quick pace. In these video cameras for home use as well, image pickup can always be precisely focused on target objects if image pickup range can be properly corrected. The mechanical realization of such correction is very difficult in view of the weight, volume and cost limitations for home-use video camera products which have been made compact (e.g., only 1 kg max.) and low in cost. Also, a mechanical control system is not in line with the general trend toward still more compact products, falling short as well of the demands of general users. Thus we foresee a probable keen demand for new image pickup devices that will permit purely electronic internal image correction, a feature not available with conventional image pickup devices.
The second problem in connection with the new requirements is that clear still images of targets cannot be obtained if either the target objects or the image pickup devices are to move around in actual use. If the solid-state image pickup devices used have very short exposure time or a comparable ultrahigh speed shutter function, as is the case with still cameras, clear still images may be obtained for all fields or frames, throughout the image pickup operation, at the intended high degree of resolution. With a method relying on conventional image pickup devices, however, which have a slow shutter speed equivalent to the unit field, or 1/60 second, degraded still image resolution is inevitable due to the picture drift accompanying the movement of the target or the image pickup device, as mentioned above. As a matter of course, the faster the movement, the more extreme the resultant degradation in resolution. As a countermeasure to this problem, a method has been proposed in which electric charges are intentionally removed during the light receiving period to shorten it. Although equivalent to increasing the shutter speed, this method has the natural consequence of reducing the sensitivity in an inverse proportion to the shutter speed. Thus, it is contrary to the general trend of development aiming at higher sensitivity and at meeting user demand, and suggests the serious problems that may be posed if it is adapted for industrial and home electronics uses.
Now, let us see in more detail how this second problem affects conventional solid-state image pickup devices used in industrial applications. When object perception is intended with movable robots and the like, a problem is posed at the stage of initial image input. In ordinary object perception technique, not an animated picture of the object but at least one clear still picture of the object is necessary, and it is common practice to perceive the object by processing such image information. Accordingly, even when either the object or the device itself is moving, clear still pictures of the object are required to maintain the desired object perception function. However, in such a case, due to the picture movement which occurs during the received light accompanying the aforementioned movements of the object or the device, image resolution input is degraded.
As described above, light reception is effected in the case of CCD and the like devices for relatively longer field periods of around 1/60 second. This causes picture drift at fairly high frequencies during light reception due to the ordinary movements of the target object or the device itself, degrading still picture resolution, which eventually lowers object perception performance. There has thus been a demand for a function that achieves clearer images via counter-correction or cancellation at far higher speeds than the 1/60 second of the image pickup device in response to unit control signals accompanying the movements of target object or the unit itself. In order to realize this function, however, control at fairly high speed is required, a response performance as high as several thousand hertz being needed to obtain perfectly still pictures. Since accomplishing the function by mechanical means is almost impossible, the need for devices which can accomplish the function electronically is anticipated. Although the function have been provided in conventional solid-state image pickup devices, such devices have been limited in application, owing to the resulting marked reduction in sensitivity and low industrial value.
Now, a look at how the second problem affects conventional image pickup devices when used in home electronics products. Some electronic still cameras have been introduced as trial products for public communication use. If existing CCD and the like image pickup devices are employed in such cameras, slippage off of still pictures is likely to occur with respect to the set exposure time, which is determined by the field period of 1/60 second, as described above. This has necessitated increasing shutter speeds by electronic means, as in the case of types used in industrial applications. To meet the requirement for clearer still pictures, a method is under review in which light receiving time is shortened by discharging photoelectric conversion charges so as to shorten the exposure time equivalently, i.e. to increase the shutter speeds. In this case, however, the problem of lowered sensitivity results, asdescribed before. This problem goes counter to the recent trend of technical development aiming at CCD's and other solid-state image pick-up devices with still higher sensitivities, as well as to user needs.
As described above, the two-dimensional solid-state image pickup device employing CCD or MOS devices have only the limited function of regularly outputting image information, within a certain area of the light receiving units, as image signals in response to reference time signals. For this reason, they cannot meet such new requirements as described above and present the problem of the limited image pickup range even in their present limited field of application. It is well anticipated that this will pose a serious problem in future as the scope of application of these elements expands. To date, however, no means of embodying workable image pickup systems, capable of meeting this requirement have been disclosed.
As another approach to this end, a "two-state image processor SI", designed as one chip, was announced in Nikkei Electronics (No. R195-216) of Dec. 19, 1983, in which complete image information is said to be tentatively stored in the digital memory, after A-D conversion, to undergo subsequent image processing for parallel movements or rotations. In this case, however, it takes 0.53.mu./bit for parallel movement of the picture alone. This means several seconds are required for parallel movement of an ordinary TV picture; thus it cannot be adapted to real time processing of continuous pictures on a TV screen, which requires 60 fields per second.
Real time processing requires an ultra-high speed processor, and is impracticable with a single-chip setup even if the memory unit is excluded. Such a scheme is as yet some time in the future; thus, the real time approach to TV signal image processing via such a digital image processor has proved to be unrealizable in the near future.